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Preface 


The  National  Audit  Office  (NAO)  is  the  supreme  audit  institution  of  the  UK  and  audits 
the  accounts  of  all  central  government  departments  and  agencies.  The  NAO  also  seeks  to 
ensure  that  the  taxpayer  receives  value  for  money  from  public  spending. 

The  NAO  commissioned  RAND  Europe  to  investigate  the  impact  of  its  value  for  money 
work  with  the  UK  Ministry  of  Defence.  In  particular,  RAND  Europe  was  asked  to  identify 
and  validate  financial  savings  that  had  been  achieved  as  a  result  of  recommendations  made 
by  the  NAO. 

This  report  provides  a  brief  outline  of  our  approach  and  a  summary  of  our  key  findings. 

RAND  Europe  is  an  independent  not-for-profit  policy  research  organisation  that  aims  to 
improve  policy  and  decision  making  in  the  public  interest,  through  research  and  analysis. 
RAND  Europe’s  clients  include  European  governments,  institutions,  NGOs  and  firms 
with  a  need  for  rigorous,  independent,  multidisciplinary  analysis.  This  report  has  been 
peer-reviewed  in  accordance  with  RAND’s  quality  assurance  standards. 

For  further  information  please  contact  Dr  Matt  Bassford 

RAND  Europe 
Westbrook  Centre 
Milton  Road 
Cambridge  CB4  1YG 
United  Kingdom 
Tel.  +44  (0)  1223  333  329 

reinfo@rand.org 
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Summary 


What  was  RAND  Europe  asked  to  do? 

The  National  Audit  Office  (NAO)  Defence  Value  for  Money  (VFM)  team  has  written 
over  35  reports  in  the  last  five  years  relating  to  the  Ministry  of  Defence  (the  Department). 
These  reports  seek  to  provide  implementable  advice  and  to  promulgate  good  practice. 

The  purpose  of  the  RAND  Europe  study  was  to  identify  which  of  these  reports  are  likely 
to  have  led  to  financial  savings  within  the  Department  and,  where  practicable,  to  quantify 
and  validate  those  savings  that  have  been  realised  by  the  Department  as  a  result  of  NAO 
recommendations. 

How  did  RAND  Europe  approach  the  study? 

Stage  1  comprised  an  initial  sift  of  all  VFM  reports  published  by  the  Defence  VFM  team 
in  the  period  2002  to  2007.  Through  a  desk  review  of  all  reports  we  identified  a  long-list 
of  NAO  reports  for  investigation. 

Stage  2  involved  a  detailed  review  of  the  NAO  reports  on  our  long-list.  We  consulted 
other  relevant  documents  and  conducted  interviews  with  NAO  personnel.  The  output  of 
Stage  2  was  a  shortlist  of  seven  VFM  reports  for  final  investigation. 

Stage  3  comprised  identification,  quantification  and  validation  of  savings  achieved  by  the 
Department  due  to  NAO  recommendations  in  selected  VFM  reports.  Here  we  conducted 
a  series  of  semi-structured  interviews  with  relevant  personnel  from  within  the  Department 
and  forensic  review  of  documentation  and  financial  information  from  the  Department. 

What  were  the  key  findings  of  the  study? 

As  a  result  of  our  study,  we  have  identified  £116.8  million  of  financial  savings  that  have 
been  achieved  by  the  Department  in  relation  to  an  NAO  study  on  optimising  stock 
holdings.  We  have  agreed  with  the  Department  that  it  is  appropriate  to  attribute  25  per 
cent  of  these  savings,  £29.8  million,  to  the  NAO  Defence  VFM  team  due  to  the 
recommendations  made  in  the  relevant  NAO  study.  In  addition,  we  have  highlighted 
savings  in  excess  of  £100  million  that  have  been  achieved  by  the  Department  which  are 
related  to  an  NAO  study  on  routine  procurement. 

The  study  also  identified  four  other  VFM  reports  that  merit  further  investigation. 
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chapter  i  Introduction 


This  chapter  provides  background  to  the  study,  outlines  the  role  of  the  National  Audit 
Office  and  gives  an  overview  of  the  study. 


Purpose  of  the  Study 

The  purpose  of  this  study  was  to  follow  up  conclusions  and  recommendations  made  by  the 
National  Audit  Office  (NAO)  which  may  have  led  to  financial  savings  for  the  Ministry  of 
Defence.  The  work  was  commissioned  by  the  NAO  in  order  to  assess  the  impact  it  had 
had  with  the  Ministry  of  Defence  (the  Department)  and  whether  or  not  the  impact  of  its 
work  with  the  Department  had  been  fully  represented  in  previous  investigations  by  the 
NAO.  In  particular,  RAND  Europe  was  asked  to  identify  any  instances  where  the 
Department  had  made  financial  savings  due  to  NAO  recommendations. 


Background 

The  NAO  is  the  supreme  audit  institution  of  the  UK  and  audits  the  accounts  of  all  central 
government  departments  and  agencies  in  addition  to  a  wide  range  of  other  public  bodies. 
The  NAO  is  independent  of  government,  with  its  budget  set  by  Parliament  and  the 
Comptroller  and  Auditor  General  working  to  Parliament  rather  than  the  government  of 
the  day.  In  addition  to  its  role  as  a  financial  auditor,  the  NAO  seeks  to  ensure  that  the 
taxpayer  receives  value  for  money  from  government  departments  and  other  public  bodies. 
To  achieve  this  objective,  the  NAO  works  with  departments  in  three  key  areas: 

•  Economy:  minimising  the  cost  of  resources  used  or  required; 

•  Efficiency:  the  relationship  between  the  output  from  goods  or  services  and  the 
resources  used  to  produce  them; 

•  Effectiveness:  the  relationship  between  the  intended  and  actual  results  of  public 
spending. 
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The  NAO  summarises  the  conclusions  and  recommendations  arising  from  this  work  in 
publicly  available  reports  which  seek  to  provide  implementable  advice  and  to  highlight 
good  practice.1 


1 .3  The  NAO's  Role  in  Improving  Value  for  Money  for  the  Taxpayer 

In  order  to  measure  the  impact  of  its  reports,  the  NAO  works  with  departments  to  identify 
financial  savings  that  have  been  achieved  as  a  result  of  NAO  recommendations.  These 
financial  savings  can  be  redeployed  within  the  departments  to  deliver  higher  quality 
outcomes,  or  can  contribute  to  public  sector  efficiency  targets.  In  fact,  the  NAO  has  an 
organisational  performance  target  that  every  £1  spent  by  the  NAO  should  result  in  savings 
of  £9  to  the  public  purse.  Consequently,  the  NAO  seeks  to  demonstrate  the  impact  its 
recommendations  have  made  by  attributing  or  ‘claiming’  a  proportion  of  the  savings  that 
are  achieved  by  departments. 

It  can  be  challenging  to  delineate  the  impact  of  the  NAO  recommendations  from  internal 
initiatives  within  departments  or  agencies,  and  potentially  from  the  work  of  the  Office  of 
Government  Commerce.  However,  the  work  of  the  NAO  can  provide  an  extra  degree  of 
focus  to  the  work  of  other  auditors  and  can  increase  the  impact  of  their  recommendations. 
To  provide  rigour  to  assessing  the  impact  of  its  work,  the  NAO  has  developed  an  internal 
guidance  framework  that  is  used  in  identifying  savings  claims.2  The  criteria  for  making  a 
claim  are  illustrated  in  Figure  1. 


Situation 

Scope  for  NAO  claim 

The  NAO  recommendation  is  the  same  as  (and 
sometimes  partly  derived  from)  earlier  ones 
contained  in  the  department’s  plans 

Only  claimable  by  the  NAO  if  work 
provides  a  fresh  impetus  to  the 
department  taking  action 

The  NAO  recommendation  is  similar  to  an  idea 
being  circulated  for  discussion  within  the 
department 

Partly  claimable  by  the  NAO  if  the 

NAO  recommendation  was 
instrumental  in  the  idea’s  being  accepted 

The  department  initiated  a  review  when  the 

NAO  disclosed  it  was  starting  a  study  in  the 
area,  and  the  department  produced  its  plans 
before  the  NAO  report  was  finalised 

Still  largely  claimable  by  the  NAO  if  the 
NAO  study  was  instrumental  in  the  idea 
being  reviewed 

Although  the  NAO  originated  the  idea  the 
normal  departmental  monitoring  process  could 
have  led  to  the  policy  change  in  response  to 
changing  circumstances 

Only  claimable  by  the  NAO  if  marginal 
benefits  were  achieved  from  earlier 
implementation 

Figure  1 :  Attributing  Impacts  to  the  NAO 


A  proportion  of  the  reports  generated  by  the  NAO  are  reviewed  by  the  Committee  of  Public  Accounts  (PAC).  The  PAC  is 
charged  with  examining  the  accounts  of  bodies  that  have  funds  voted  for  by  Parliament,  and  it  achieves  this  task  by  taking 
evidence  from  various  government  departments  and  public  bodies  which  are  responsible  for  managing  public  funds. 

2  Are  YOU  making  an  impact ?  -  A  guide  to  planning  for,  measuring  and  recording  the  financial  impact  of  audit  work ,  National 
Audit  Office 
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Consequently,  the  NAO  is  required  to  follow  up  with  departments  on  financial  savings 
that  might  be  attributed  to  the  NAO.  Once  potential  savings  claims  have  been  identified 
they  must  be  agreed  with  the  department.  The  internal  guidance  states  that:3 

A  discussion  of  attribution  to  the  NAO  is  usually  part  of  the  agreement  process  for  each 
financial  impact.  Normal  practice  is  to  agree  with  the  department  or  agency  concerned  a 
percentage  of  the  identified  saving  that  could  reasonably  be  attributed  to  NAO  intervention. 

In  addition  to  obtaining  agreement  from  the  department,  potential  savings  claims  must 
also  satisfy  internal  audit  processes  and  other  checks  and  balances.4  It  is  a  prerequisite  of 
all  savings  claims  that  an  unequivocal  link  between  actual  financial  savings,  departmental 
action  and  NAO  recommendation  is  demonstrated  and  supported  by  suitable  evidence. 


1 .4  The  NAO's  Work  with  the  Ministry  of  Defence 

The  NAO  is  structured  around  different  Value  for  Money  (VFM)  teams,  most  of  which 
are  aligned  with  a  government  department.  Our  study  was  focused  on  the  work  of  the 
Defence  VFM  team,  which  considers  the  Ministry  of  Defence. 

The  Department  is  a  large  and  complex  government  department  with  an  annual  budget  of 
nearly  £34  billion  and  fixed  assets  of  approximately  £90  billion.5  The  Defence  VFM  team 
in  the  NAO  undertakes  studies  that  assess  the  value  for  money  spending  within  the 
Department.  This  includes  the  ability  of  the  Department  to  support  current  operations, 
major  equipment  acquisitions,  business  change  programmes  and  support  for  Armed  Forces 
personnel.  In  addition,  the  effectiveness  and  efficiency  of  underpinning  processes  and 
infrastructure  are  assessed. 

The  Defence  VFM  team  has  published  around  33  reports  in  the  last  five  years  directly 
related  to  the  Department,  which  are  detailed  at  Appendix  A.  However,  despite  the 
substantial  attention  given  to  the  Department  over  this  time,  the  Defence  VFM  team  has 
only  been  able  to  identify  and  claim  relatively  limited  savings  that  can  be  attributed  to 
their  work. 

The  team  can  claim  savings  achieved  by  the  Department  within  three  cost  types,  namely: 
capital  expenditure  (referred  to  as  CDEL  or  Capital  Departmental  Expenditure  Limit); 
operating  costs  (referred  to  as  RDEL  or  Resource  Departmental  Expenditure  Limit);  and 
indirect  costs  (referred  to  as  IRDEL  or  Indirect  Resource  Departmental  Expenditure 
Limit).  Examples  of  cost  centres  within  which  savings  can  be  achieved  are  shown  in  Figure 
2. 


3  ibid. 

4  Ibid. 

5  Ministry  of  Defence  Annual  Report  and  Accounts  2007-2008,  Ministry  of  Defence 
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Cost  type 

Examples 

Capital  Expenditure  (CDEL) 

Infrastructure 

Equipment 

Estate 

Operating  Costs  (RDEL) 

People 

Service  provisions 

Running  costs 

Consumables 

Operational  support 

Indirect  Resource  Costs  (IRDEL) 

Cost  of  capital 

Depreciation 

Figure  2:  Classification  of  MOD  Cost  Types 


1 .5  Objectives  of  this  Study  and  Report  Structure 

The  NAO  Defence  VFM  team  commissioned  RAND  Europe  to  assess  whether  or  not  the 
impact  of  their  work  with  the  Department  had  been  fully  represented  in  the  savings  claims 
that  had  been  identified  by  the  NAO.  In  particular,  RAND  Europe  was  asked  to  identify 
the  potential  for  attributing  additional  financial  savings  where  the  Department  had  made 
financial  savings  due  to  NAO  recommendations. 

The  specific  study  objectives  for  RAND  Europe  were  to: 

•  Review  the  conclusions  and  recommendations  made  by  the  Defence  VFM  team; 

•  Identify  potential  financial  savings  within  the  MOD  resulting  from  any 
implementation  of  these  conclusions  and  recommendations; 

•  Validate  the  identified  savings  with  the  Department  and  determine  what  proportion 
can  be  attributed  to  the  recommendations  and  conclusions  of  the  NAO. 

This  report  provides  a  brief  summary  of  our  approach,  results  and  conclusions.  The 
structure  of  this  document  is  as  follows: 

•  Chapter  2:  Results  This  chapter  summarises  our  approach  and  the  outcome  of  our 
assessment  of  over  35  reports  published  by  the  NAO  in  relation  to  the  Department. 

•  Chapter  3:  Conclusions  and  Scope  for  Future  Work  This  chapter  provides  a  short 
summary  of  lessons  learned  from  our  study  and  outlines  potential  further  work. 


4 


RAND  Europe 


Chapter  2  -  Results 


CHAPTER  2  ReSljItS 


This  chapter  provides  a  summary  of  our  approach  and  the  key  findings  of  the  study. 


2. 1  Summary  of  Methodology 

The  primary  objective  of  the  study  was  to  identify  and  quantify  any  financial  savings  that 
had  been  achieved  by  the  Department  as  a  result  of  NAO  recommendations.  However, 
within  the  time  and  resource  constraints  of  our  study  it  was  not  possible  to  conduct  a 
forensic  examination  of  every  report  published  by  the  Defence  VFM  team  since  2002. 
Consequently,  we  developed  a  three-stage  approach  to  identify  those  reports  that  were 
most  likely  to  have  led  to  financial  savings  within  the  Department. 

•  Stage  1  comprised  an  initial  sift  of  all  VFM  reports  published  by  the  Defence  VFM 
team  in  the  period  2002  to  2007.  Through  a  desk  review  of  all  reports  we  identified  a 
long-list  of  NAO  reports  for  investigation. 

•  Stage  2  involved  a  detailed  review  of  the  NAO  reports  on  the  long-list  identified  in 
Stage  1.  We  consulted  other  relevant  documents  and  conducted  interviews  with  NAO 
personnel.  The  output  of  Stage  2  was  a  shortlist  of  seven  VFM  reports  for  final 
investigation. 

•  Stage  3  comprised  identification,  quantification  and  validation  of  savings  achieved  by 
the  Department  due  to  NAO  recommendations  in  selected  VFM  reports.  Here  we 
conducted  a  series  of  semi-structured  interviews  with  relevant  personnel  from  within 
the  Department  and  forensic  review  of  documentation  and  financial  information  from 
the  Department. 

A  diagrammatic  summary  of  the  methodology  is  included  at  Figure  6. 


2.2  Selection  of  NAO  Reports 

In  the  period  between  2002  and  2007,  the  Defence  VFM  team  in  the  NAO  has  published 
some  35  reports  relating  to  the  Department,  most  of  which  have  also  been  reported  on  by 
the  Committee  of  Public  Accounts.  These  reports  can  be  categorised  into  three  groups: 

•  Major  Project  Reports  (MPR)  The  MPR  is  published  annually  to  report  performance  of 
the  20  highest  value  defence  procurement  projects  against  cost,  time  and  quality. 
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These  reports  were  not  considered  in  our  study  as  their  focus  is  on  assessing  progress 
rather  than  providing  specific  efficiency  recommendations. 

•  'Gold  Standard'  reports  In  order  to  help  the  Department  improve  its  performance,  the 
NAO  has  also  published  studies  assessing  project  control  and  effective  contracting. 
Both  types  of  study  compare  current  defence  performance  to  a  theoretical  ‘gold 
standard’.  The  recommendations  in  these  reports  are  intended  to  bring  improvements 
in  defence  acquisition  performance  to  help  ensure  all  defence  projects  routinely  adopt 
practices  closer  to  this  ‘gold-standard’.  However,  the  specific  impacts  of  this  work  are 
difficult  to  track  individually  due  to  the  strategic  nature  of  the  recommendations. 
Consequently,  these  reports  were  not  considered  in  our  study. 

•  Value  for  Money  (VFM)  reports  These  reports  examine  the  economy,  efficiency  and 
effectiveness  of  spending  within  the  Department.  VFM  reports  typically  evaluate  major 
acquisition/change  programmes  or  other  thematic  areas.  A  key  component  of  all  VFM 
reports  is  a  set  of  specific  recommendations  designed  to  improve  economy,  efficiency 
and/or  effectiveness;  consequently,  VFM  reports  can  often  lead  to  financial  savings 
within  the  Department.  As  a  result,  we  focused  on  the  23  VFM  reports  published  by 
the  Defence  team  between  2002  and  2007  during  our  study.  In  addition  we  also 
considered  two  other  reports,  which  had  made  recommendations  that  have  been 
adopted  by  the  Department,  making  a  total  of  23  VFM  reports.6 

A  full  list  of  reports  is  presented  at  Appendix  A. 


2.3  Stage  1:  Initial  Sift  of  VFM  Reports 

During  Stage  1  we  conducted  an  initial  sift  of  all  VFM  reports  published  by  the  Defence 
VFM  team  in  the  period  2002  to  2007.  The  purpose  of  this  research  stage  was  to  identify 
which  of  the  reports  published  by  the  NAO  were  most  likely  to  have  resulted  in  significant 
financial  savings  within  the  Department.  The  output  of  this  research  stage  was  a  long-list 
of  those  reports  that  could  potentially  have  resulted  in  significant  cost  savings  within  the 
Department. 

The  first  step  in  this  initial  sift  was  to  exclude  those  reports  for  which  the  NAO  had 
already  agreed  a  financial  saving  with  the  Department.  Consequently,  reports  relating  to 
the  management  of  defence  research  and  technology  and  to  compensation  claims  were 
excluded  from  further  study.7 

The  second  step  was  to  assess  each  report  against  two  criteria,  designed  to  highlight  those 
reports  which  were  most  likely  to  have  resulted  in  quantifiable  savings: 

•  Does  the  report  identify  quantifiable  savings  or  one  which  is  well  bounded  and  able 
to  be  calculated?  We  only  pursued  reports  that  satisfied  this  criterion. 


6  The  Wider  Markets  Initiative,  2006,  National  Audit  Office,  and  Improving  the  Efficiency  of  the  Procurement  of  Routine  Items, 
1 997,  National  Audit  Office 

7  The  Management  of  Defence  Research  and  Technology,  2004,  National  Audit  Office,  and  Compensation  claims,  2003, 
National  Audit  Office 
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•  Is  this  an  area  where  the  Department  is  likely  to  have  achieved  significant  efficiency 
savings?  We  only  pursued  reports  for  which  it  was  reasonable  to  assume  that  savings 
in  excess  of  £1  million  could  have  been  achieved. 

The  results  of  our  assessment  against  these  criteria  are  illustrated  in  Figure  3  and  4.  A 

simple  ‘traffic  light’  was  employed  to  indicate  this  assessment. 

•  Figure  3  highlights  those  reports  that  were  thought  to  have  the  most  potential  for 
quantifiable  cost  savings,  and  were  worthy  of  inclusion  on  our  long-list  of  reports  for 
further  investigation. 

•  Figure  4  contains  the  reports  which  were  assessed  not  to  offer  a  high  potential  for 
quantifiable  savings,  and  were  therefore  excluded  from  our  long-list  of  reports. 


NAO  report8 

Year  of 
publication 

Quantifiable  savings? 

Potential  size  of  savings 

Transforming  logistics  support  for 
fast  jets 

2007 

Progress  in  Reducing  Stocks 

2002 

Improving  the  Efficiency  of  the 
Procurement  of  Routine  Items 9 

1997 

The  Wider  Markets  Initiative 

2006 

Recruitment  and  Retention  in  the 
Armed  Forces 

2006 

Improving  Service  Delivery:  The 
Veterans  Agency 

2003 

Major  Repair  and  Overhaul  of 
land  Equipment 

2002 

The  construction  of  nuclear 
submarine  facilities  at  Devonport 

2002 

Managing  the  Defence  Estate: 
Quality  and  Sustainability 

2007 

Managing  the  Defence  Estates 

2005 

Battlefield  Helicopters 

2004 

The  Joint  Services  Command 
and  Staff  College 

2002 

Figure  3:  Initial  Sift  -  Long-list  of  Reports  with  Potential  for  Cost  Savings 


8  Reports  detailed  in  Appendix  A 

9  Published  outside  time  range  of  other  reports,  but  included  on  the  basis  of  its  high  potential  for  savings 
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NAO  report" 


Year  of 
publication 


Quantifiable  savings? 


Potential  size  of  savings 


Reserve  Forces 


2006 


Quality  of  Housing  Services  to 
Service  Families  Overseas 


2005 


The  Rapid  Procurement  of 
Capability  to  Support  Operations 


2004 


Helicopter  Logistics 


2002 


The  privatisation  ofQinetiQ 


2007 


Leaving  the  Services 


2007 


Operation  TELIC  -  United 
Kingdom  Military  Operations  in 
Iraq 


2003 


Progress  in  Combat  Identification 


2006 


Delivering  digital  tactical 
communications  through  the 
Bowman  CIP programme 


2006 


Assessing  and  Reporting  Military 
Readiness 


2005 


Exercise  SaifSareea  II 


2002 


Building  an  Air  Manoeuvre 
Capability:  The  Introduction  of 
the  Apache  Helicopter 


2002 


Combat  Identification 


2002 


Figure  4:  Initial  Sift  -  Reports  Excluded  due  to  Limited  Potential  for  Cost  Savings 


10  Reports  detailed  in  Appendix  A 
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2.4  Stage  2:  Constructing  a  Short-list 

The  purpose  of  Stage  2  was  to  produce  a  short-list  of  5  to  10  reports  for  final  investigation. 
This  research  stage  involved  a  more  detailed  review  of  the  NAO  reports  on  our  long-list 
and  other  relevant  documents  including  PAC  hearings,  Treasury  minutes  and 
Parliamentary  debates.  We  also  conducted  interviews  with  NAO  personnel  who  had  been 
involved  in  preparation  of  the  relevant  reports. 

The  reports  on  the  long-list  generated  through  the  initial  sift  were  assessed  against  two 
further  criteria: 

•  Is  it  possible  to  construct  a  counterfactual  case  for  the  report?  In  order  to  quantify 
financial  savings  achieved  by  the  Department,  it  was  necessary  to  compare  the  savings 
with  a  theoretical  counterfactual  case.  This  ex  post  counterfactual  was  necessary  to 
establish  what  costs  would  have  been  had  the  Department  not  implemented  the  NAO 
recommendations.  The  relevant  NAO  guidance  states  that:" 

...  [we]  should  attempt  to  quantify  the  efficiency  gain  that  has  been  realised.  It  is  important  to 
note,  hoivever,  that  doing  so  does  not  involve  a  simple  ‘before  and  after’  comparison.  Rather,  the 
appropriate  comparison  is  against  a  counterfactual  (unobserved)  situation,  namely  the  situation 
that  ivould  have  obtained  had  the  recommendation  not  been  acted  upon. 

Two  reports  on  our  original  long-list  were  excluded  due  to  the  challenges  of 
constructing  a  counterfactual  case:  those  relating  to  the  Joint  Services  Command  and 
Staff  College,  and  to  the  construction  of  nuclear  submarine  facilities  at  Devonport. 

•  Would  any  savings  claims  conflict  with  ongoing  NAO  work?  In  discussion  with  the 
NAO  it  was  agreed  that  our  study  should  exclude  any  recommendations  that  were  the 
subject  of  ongoing  NAO  work.  Consequently  three  reports  were  excluded:  those 
relating  to  Defence  Estates  and  the  report  on  recruitment  and  retention  of  Armed 
Forces  personnel.  These  reports  may  be  worthy  of  further  investigation  by  the  NAO  at 
a  future  point  in  time. 


2.5  Stage  3:  Detailed  Investigation  of  Selected  Reports 

The  objective  of  the  third  stage  of  our  study  was  to  identify  and  quantify  potential  savings 
achieved  by  the  Department  due  to  NAO  recommendations  and  to  validate  these  savings 
with  the  Department. 

During  this  research  stage  we  conducted  a  series  of  semi-structured  interviews  with  relevant 
personnel  from  within  the  Department,  a  thorough  review  of  internal  documents  provided 
by  the  Department  and  analysed  financial  data  provided  by  the  Department. 

Due  to  the  relatively  resource-intensive  nature  of  this  research  phase  it  was  not  possible  to 
investigate  all  of  the  reports  on  our  short-list  in  forensic  detail.  Consequently,  we  selected 


11  Are  YOU  making  an  impact ?  -  A  guide  to  planning  for,  measuring  and  recording  the  financial  impact  of  audit  work,  National 
Audit  Office 


9 


RAND  Europe 


Chapter  2  -  Results 


three  reports  for  detailed  investigation  on  the  basis  of  the  criteria  addressed  in  Stages  1  and 
2.  The  three  reports  selected  were: 

•  The  Wider  Markets  Initiative; 

•  Progress  in  Reducing  Stocks; 

•  Improving  the  Efficient  of  the  Procurement  of  Routine  Items. 

The  results  of  our  detailed  investigation  are  presented  in  the  following  sections. 

We  did  not  conduct  a  detailed  investigation  of  the  other  reports  on  our  short-list  in  this 
study.  However,  an  evaluation  of  the  prospect  for  further  investigation  of  the  following 
reports  is  provided  in  Chapter  3. 

•  Major  Repair  and  Overhaul  of  Land  Equipment; 

•  Transforming  logistics  support  for  fast  jets; 

•  Battlefield  Helicopters; 

•  Improving  Service  Delivery:  The  Veterans  Agency. 

2.5.1  The  Wider  Markets  Initiative 

The  Wider  Markets  Initiative  (2006)  report  was  a  cross-government  study  that  made  wide- 
ranging  recommendations  on  maximising  income  from  public  assets.  The  report 
highlighted  a  number  of  examples  of  good  practice  that  had  been  observed  in  the  Ministry 
of  Defence,  but  noted  that  there  was  scope  for  improvement. 

During  a  semi-structured  interview,  the  Director  of  Finance  Policy  within  the  Department 
recognised  that  the  NAO  report  had  helped  maintain  progress  on  wider  markets.  However, 
the  Department  does  not  keep  financial  records  in  sufficient  granularity  to  quantify  and 
attribute  the  level  of  income  from  wider  market  activities.  Without  these  financial  records, 
it  was  not  possible  to  quantify  the  financial  impact  of  the  NAO  report  on  wider  markets. 

2.5.2  Progress  in  Reducing  Stocks 

The  focus  of  the  NAO  report  Progress  in  Reducing  Stocks  (2002)  was  to  identify  areas  where 
the  Department  had  undertaken  specific  actions  to  reduce  stocks  and  to  highlight  areas 
where  there  was  further  potential  for  optimising  the  stocks  held  by  the  Department.  In 
particular,  the  NAO  report  recommended  that  the  Department  should  assess  the  scope  for 
stock  reductions  and  take  full  account  of  the  large  volumes  of  slow  moving  stock, 
particularly  consumables. 

Interviews  with  staffs  responsible  for  stock  optimisation  within  the  Department  confirmed 
the  impact  that  the  NAO  report  had  had  in  providing  fresh  impetus  to  stock  optimisation. 
In  particular,  the  NAO  recommendations  were  highlighted  in  internal  guidance  that  was 
issued  to  all  IPTs  (Integrated  Project  Teams)  involved  in  stock  management  regarding  the 
potential  for  disposal  of  obsolete,  obsolescent,  slow  or  non-moving  stock.12  This  guidance 
stated  that: 

The  Defence  Logistics  Organisation  (DLO)  has  over  recent  years  been  criticised  by  the  National 
Audit  Office  (NAO)  and  the  Public  Accounts  Committee  regarding  excessive  inventory 


12  //i'/'  Inventory  Review  —  Depot  Ratiotialisation  Programme ,  SCS/Inventory  Optimisation! DSD  A  Version  6.0,  21 
July  2006 
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holdings.  Although  the  DLO  has  made  significant  progress  in  reducing  inventory  levels  no 
longer  required  for  business  purposes  it  is  widely  recognised  in  the  organisation  that  there  is  still 
significant  scope  for  further,  targeted  reduction  and  process  improvement. 

The  NAO  state  in  their  Report  on  Progress  in  Reducing  Stocks  (HC  898  dated  June  2002), 
that  ‘reducing  stocks  will  lead  to  significant  financial  savings  and  contribute  to  depot 
rationalisation  ’.  This  assertion  still  holds  good  today. 

Interviews  were  also  conducted  with  senior  staffs  from  the  Chief  Management  Accounting 
(CMA)  team  within  the  Defence  Equipment  &  Support  (DE&S)  organisation.  These 
interviews  and  the  following  discussion  confirmed  that  stock  holdings  had  been  reduced 
since  the  internal  guidance  and  that  these  reductions  were  due,  in  part,  to  the  NAO 
recommendations.  During  FY07/08,  the  Department  had  achieved  audited  financial 
savings  of  £1 16.8  million  as  a  result  of  stock  optimisation  relating  to  defence  equipment 
and  support. 

On  the  basis  of  discussions  with  the  CMA  team,  and  comparison  with  previous  NAO 
savings  claims,  we  assessed  that  25%  of  the  £116.8  million  savings  achieved  by  the 
Department  in  FY07/08  could  be  attributed  to  the  NAO  recommendations  and 
conclusions  relating  to  stock  management. 

A  savings  claim  of  £29.2  million  has  been  agreed  between  the  NAO  and  the  Department. 
Further  details  on  this  savings  claim  are  presented  at  Appendix  B. 

2.5.3  Improving  the  Efficiency  of  the  Procurement  of  Routine  Items 

The  Improving  the  Efficiency  of  the  Procurement  of  Routine  Items  report  (1997)  assessed 
routine  procurement  within  the  Department.  The  report  made  a  number  of 
recommendations  relating  to  improvements  in  efficiency  or  effectiveness,  including  the 
introduction  of  corporate  purchasing  cards  to  reduce  process  costs  and  improve 
management  information. 

Following  the  publication  of  the  report,  the  Department  has  enthusiastically  introduced 
the  Government  Procurement  Card  (GPC)  and  published  guidance  on  its  use.13  The  use 
of  the  GPC  has  led  to  efficiency  savings  in  the  Department,  by  reducing  transaction  and 
audit  costs.  These  savings  were  quantified  by  the  Department  in  a  recent  response  to  a 
parliamentary  question:^ 

Since  its  implementation  in  the  MOD,  the  GPC  has  achieved  total  process  and  administration 
cost  savings  of£l  01, 894, 044  based  on  the  NAO  savings  figure  of £28  per  transaction. 

RAND  Europe  has  recommended  that  the  NAO  pursue  the  opportunity  for  a  savings 
claim  with  the  Department  based  on  these  overall  savings.  The  breakdown  of  these  savings 
over  time  is  illustrated  in  Figure  5. 


13  JSP  332,  Low  Value  Purchasing  Manual,  May  2006 

14  Bob  Ainsworth,  Minister  of  State  for  the  Armed  Forces,  Hansard  28  Apr  2008:  Column  56W  —  Departmental  Credit 
Cards 
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Financial  year 

Savings  (£) 

99/00 

61,824 

00/01 

844,340 

01/02 

1,716,288 

02/03 

6,376,300 

03/04 

11,255,440 

04/05 

17,556,168 

05/06 

20,005,188 

06/07 

21,121,212 

07/08 

22,957,284 

Total 

101,894,044 

Figure  5:  Financial  Savings  Achieved  by  Department  due  to  Government  Procurement  Card 


2.6  Summary 

This  chapter  has  summarised  the  main  elements  of  our  approach  and  the  most  significant 
outcomes  of  our  study.  We  constructed  a  three-stage  ‘sift’  to  develop  a  short-list  of  NAO 
reports  that  were  most  likely  to  have  had  a  quantifiable  financial  impact  within  the 
Department. 

As  a  result  of  a  detailed  investigation  of  three  VFM  reports,  we  have  identified  financial 
savings  relating  to  stock  optimisation  totalling  £116.8  million.  We  have  agreed  with  the 
Department  that  it  is  appropriate  to  attribute  25%  of  these  savings,  £29.8  million,  to  the 
NAO  Defence  VFM  team.  In  addition,  we  have  highlighted  savings  in  excess  of  £100 
million  that  are  related  to  an  NAO  study  on  routine  procurement. 

The  key  elements  of  our  study  are  summarised  in  Figure  6. 
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Is  it  possible  to  construct  a 
counterfactual  case  for  the 
report? 


Report  excluded  by  RAND,  as  no 
counterfactual  can  be  constructed 

•  The  Joint  Sen/ices  Command 
and  Staff  College 
•The  construction  of  nuclear 
submarine  facilities  atDevonport 

_ 


Would  any  savings  claims 
conflict  with  ongoing  NAO 
work? 


Reports  excluded  by  NAO  as  it 
would  overlap  with  current  work 

Yes  -Managing  the  Defence  Estates 

- ->  • Managing  the  Defence  Estates: 

Quality  and  Sustainability 
•Recruitment  and  Retention  in 
the  Armed  Forces 


Reports  investigated  in  detail 

•  The  Wider  Markets  Initiative 
•Progress  in  Reducing  Stocks 
•Improving  the  Efficiency  of  the 
Procurement  of  Routine  Items 


Reports  not  investigated  in  detail 

•Major  Repair  and  Overhaul  of  Land  Equipment 
•  Transforming  logistics  support  for  fast  jets 
•Battlefield  Helicopters 

•Improving  Service  Delivery:  The  Veterans  Agency 


Figure  6:  Summary  of  Methodology  and  Down-Selection  of  Reports 
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chapter  3  Conclusions  and  Scope  for  Future  Work 


This  chapter  presents  a  summary  of  lessons  learnt  from  this  study  and  identifies  those 
NAO  reports  which  offer  scope  for  further  investigation. 


3.1  Conclusions 

As  discussed  earlier  in  this  report,  the  NAO  Defence  VFM  team  has  a  broad  remit  which 
guides  their  work  with  the  Department.  Reports  will  not  always  have  a  focus  on  efficiency 
or  the  identification  of  financial  savings.  However,  where  the  impact  of  the  report  is  best 
expressed  in  terms  of  financial  savings  achieved  by  the  Department,  there  appears  to  be  a 
number  of  characteristics  associated  with  successful  savings  claims.  Whilst  not  exhaustive, 
our  experience  of  working  with  the  NAO  and  the  Department  on  this  study  suggests  the 
following  areas  of  good  practice  for  the  VFM  team: 

•  Develop  specific,  actionable  recommendations  for  the  Department.  Quantifying 
financial  benefits  realised  by  the  Department  as  a  consequence  of  NAO  reports  is  only 
realistic  when  these  reports  contain  specific  recommendations.  In  many  cases  we  found 
that  it  was  very  challenging  to  track  progress  against  recommendations  contained  in 
the  VFM  reports. 

•  Identify  key  stakeholders  in  the  Department.  It  is  likely  that  recommendations  made 
by  the  NAO  are  most  likely  to  be  implemented  where  there  is  a  clear  Department 
owner(s)  the  action.  In  addition,  this  enables  the  NAO  to  review  progress  with  the 
Department,  which  provides  the  opportunity  to  track  financial  benefits. 

•  Develop  a  collaborative  approach.  RAND  Europe  was  able  to  interact  with  the 
Department  and  the  NAO  objectively  and  develop  a  collaborative  approach.  In  simple 
terms,  the  Department  and  the  NAO  have  a  common  interest  in  quantifying  and 
validating  financial  savings  achieved  by  the  Department  that  relate  to  NAO  reports. 

•  Build  evidence  base  against  sound  counterfactual.  One  key  element  of  quantifying 
savings  claims  for  the  NAO  is  to  obtain  robust  financial  data  and  compare  it  with  a 
sound  counterfactual  case.  The  NAO  should  only  conduct  a  detailed  investigation 
following  a  critical  assessment  of  whether  data  can  be  obtained  and  whether  a  robust 
counterfactual  can  be  constructed. 
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3.2  Scope  for  Further  Investigation 

There  are  four  NAO  reports  that  met  the  criteria  we  developed,  but  that  it  was  not  possible 
to  investigate  within  the  current  study.  A  brief  summary  of  each  of  these  reports  is 
presented  with  a  qualitative  assessment  of  the  scope  for  further  investigation.  The  reports 
are  ordered  in  terms  of  priority  for  any  follow  up  work. 

3.2.1  Transforming  Logistics  Support  for  Fast  Jets  (2007) 

This  report  presents  a  number  of  specific  recommendations  on  improving  the  efficiency  of 
support  arrangements  for  fast  jets.  In  particular:  maximising  the  benefits  of  contracting  for 
availability  and  improving  internal  organisational  efficiency.  The  primary  reason  why  this 
report  was  not  considered  in  the  current  study  was  its  recent  publication  date.  In  our 
assessment,  this  report  is  the  most  likely  of  those  shortlisted  to  have  led  to  significant 
financial  savings  within  the  Department.  We  recommend  that  the  NAO  prioritises  this 
report  for  further  investigation. 

3.2.2  Battlefield  Helicopters  (2004) 

This  report  examined  the  progress  made  by  the  Department  since  the  inception  of  the 
Joint  Helicopter  Command  (JHC)  in  1999.  The  report  identified  areas  in  which  further 
efficiency  savings  could  be  achieved  by  the  Department.  In  our  assessment,  the  specific 
recommendations  relating  to  streamlining  joint  initial  pilot  training  and  to  harmonising 
airworthiness  across  the  three  services  are  likely  to  have  contributed  to  financial  savings. 
We  recommend  that  the  NAO  works  with  the  JHC  to  assess  progress  against  these 
recommendations  and  to  quantify  financial  savings  that  have  been  realised  as  a  result. 

3.2.3  Major  Repair  and  Overhaul  of  Land  Equipment  (2002) 

This  report  recommends  a  number  of  improvements,  primarily  relating  to  contractor 
logistic  support  and  increasing  the  amount  of  work  that  is  competed  and  contracted  out. 
In  our  assessment,  it  is  likely  that  these  recommendations  will  have  led  to  tangible  financial 
savings  to  the  Department.  However,  these  might  prove  challenging  to  quantify  given  the 
time  elapsed  since  the  report  organisational  restructuring  that  has  occurred  since  the 
report:  particularly  the  merger  of  the  Army  Base  Repair  Organisation  with  the  Defence 
Aviation  Repair  Agency  in  April  2008  to  form  the  Defence  Support  Group.  We  suggest 
that  the  NAO  works  with  the  Defence  Support  Group  to  assess  to  what  extent  the 
recommendations  in  the  report  have  been  implemented. 

3.2.4  Improving  Service  Delivery:  The  Veterans  Agency  (2003) 

This  report  considered  the  efficiency  and  effectiveness  of  service  delivery  by  the  Veterans 
Agency,  which  was  responsible  for  administering  the  War  Pensions  Scheme.  The  report 
recommends  that  the  cost  of  processing  claims  should  be  monitored,  which  may  have 
driven  efficiencies  in  service  delivery  and  led  to  financial  savings.  However,  given  the 
relatively  small  operating  costs  of  the  Agency  (£35  million  in  2003)  it  is  unlikely  that 
major  cost  savings  would  have  been  possible  while  maintaining  the  same  levels  of  service.  It 
should  be  noted  that  since  publication  of  the  NAO  report,  the  Veterans  Agency  has  been 
merged  with  the  Armed  Forces  Administration  Agency  to  form  the  Service  Personnel  and 
Veterans  Agency.  Given  the  relatively  small  size  of  the  potential  savings,  and  the  additional 
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challenges  presented  by  the  merger,  we  recommend  that  this  report  should  be  the  lowest 
priority  for  the  NAO  to  follow  up. 
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Appendix  A:  NAO  Reports  Investigated 


The  following  table  provides  a  list  of  NAO  reports  published  by  the  Defence  VFM  team 
that  were  investigated  in  this  study. 


NAO  report 

Year  of 
publication 

Ministry  of  Defence:  Major  Projects  Report  2007 

2007 

The  privatisation  ofQinetiQ 

2007 

Leaving  the  Services 

2007 

Transforming  logistics  support for  fast  jets 

2007 

Managing  the  Defence  Estate:  Quality  and  Sustainability 

2007 

Ministry  of  Defence:  Major  Projects  Report 2006 

2006 

Recruitment  and  Retention  in  the  Armed  Forces 

2006 

Delivering  digital  tactical  communications  through  the  Bowman  CIP 
programme 

2006 

Using  the  contract  to  maximise  the  likelihood  of  successful  project 

outcomes 

2006 

Reserve  Forces 

2006 

Progress  in  Combat  Identification 

2006 

Driving  the  Successful  Delivery  of  Major  Defence  Projects:  Effective 
Project  Control  is  a  Key  Factor  in  Successjiil  Projects 

2006 

Ministry  of  Defence  Major  Projects  Report  2005 

2005 

Assessing  and  Reporting  Military  Readiness 

2005 

Managing  the  Defence  Estate 

2005 

Quality  of  Housing  Services  to  Service  Families  Overseas 

2005 

The  Rapid  Procurement  of  Capability  to  Support  Operations 

2004 

Ministry  of  Defence:  Major  Projects  Report  2004 

2004 

Battlefield  Helicopters 

2004 

The  Management  of  Defence  Research  and  Technology 

2004 

Ministry  of  Defence:  Major  Projects  Report  2003 

2003 

Operation  TELIC  -  United  Kingdom  Military  Operations  in  Iraq 

2003 
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Compensation  claims 

2003 

Through  Life  Management 

2003 

Improving  Service  Delivery:  The  Veterans  Agency 

2003 

The  Construction  of  Nuclear  Submarine  Facilities  at  Devonport 

2002 

Ministry  of  Defence:  Major  Project  Reports  2002 

2002 

Building  an  Air  Manoeuvre  Capability:  The  Introduction  of  the  Apache 
Helicopter 

2002 

Exercise  SaifSareea  II 

2002 

Progress  in  Reducing  Stocks 

2002 

Helicopter  Logistics 

2002 

Major  Repair  and  Overhaul  of  Land  Equipment 

2002 

Lmplementation  ofLntegrated  Project  Teams 

2002 

Combat  Ldentification 

2002 

The  Joint  Services  Command  and  Staff  College 

2002 

Lmproving  the  Efficiency  of  the  Procurement  of  Routine  Ltems 

1997 

In  addition  we  also  considered  the  following  report  that  was  published  by  the  Cross 
Government  and  Public  Administration  team: 


The  Wider  Markets  Initiative 


2006 
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Stock  Optimisation 


The  following  table  provides  the  evidence  upon  which  the  NAO  has  agreed  a  savings  claim 
with  the  Department,  in  relation  to  the  2002  report  Progress  in  Reducing  Stocks. 


This  is  based  on  the  standard  NAO  form  for  saving  claims  as  presented  in  the  NAO 
guidance  document  Are  YOU  making  an  impact ?  -  A  guide  to  planning  for,  measuring  and 
recording  the  financial  impact  of  audit  work. 


Report/ Account  title: 

Ministry  of  Defence,  Progress  in  Reducing  Stocks,  Report 
by  the  Comptroller  and  Audit  General,  HC  898  Session 
2001-2002:  20  June  2002 

Summary  of  Impact: 

The  NAO  report  Progress  in  Reducing  Stocks  made  the 
following  recommendations: 

•  The  Department  recognises  that  whilst  surplus  stocks 
remain  on  the  books,  they  represent  a  drain  on  resources 
which  could  otherwise  be  used  to  help  make  good  current 
capability  gaps  or  fund  future  procurements.  Much  effort  is 
being  expended  on  tackling  these  issues.  We  have  identified 
examples  of  good  practice  as  well  as  our  oivn  suggestions  for 
areas  of  future  work  that  should  help  the  Department 
dispose  of  significant  volumes  of  stocks. 

o  The  Department  should  revisit  its  stock  reduction 

targets  and  ensure  that  these  take  full  account  of  the 
large  volume  of  slow  moving  stock,  particularly  the 
high  volumes  of  consumable  items 

o  In  setting  targets  the  Department  should  clearly 

define  the  target,  establish  robust  baselines  and  ensure 
consistency  in  measuring  achievements. 

These  recommendations  have  been  implemented  by  the 
MOD.  As  noted  in  IPT  inventory  Review  —  Depot 
Rationalisation  Programme,  SCS/Inventory 

Optimisation/DSDA  Version  6. 0,  21  Jidy  2006: 

•  The  Defence  Logistics  Organisation  (DLO)  has  over  recent 
years  been  criticised  by  the  National  Audit  Office  (NAO) 
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and  the  Public  Accounts  Committee  regarding  excessive 
inventory  holdings.  Although  the  DLO  has  made 
significant  progress  in  reducing  inventory  levels  no  longer 
required  for  business  purposes  it  is  ividely  recognised  in  the 
organisation  that  there  is  still  significant  scope  for  further, 
targeted  reduction  and  process  improvement. 

•  The  NAO  state  in  their  Report  on  Progress  in  Reducing 

Stocks  (HC  898  dated  June  2002),  that  ‘reducing  stocks 
will  lead  to  significant  financial  savings  and  contribute  to 
depot  rationalisation  ’.  This  assertion  still  holds  good  today. 

Since  this  MOD  publication,  three  sites  have  been,  or 
are  being,  closed,  to  further  optimise  stock  holdings. 

Following  discussions  with  the  RAND  Europe  team, 
the  Department  identified  overall  savings  due  to  stock 
optimisation  amounting  to  £116.8  million.  This 
comprises  RDEL,  IRDEL  and  CDEL  savings  of  £5.7 
million,  £1 10.6  million  and  £0.5  million,  respectively. 

It  was  agreed  with  the  Department  that  25%  of  these 
savings  (£29.2  million)  could  be  attributed  to  the  NAO 
due  to  the  extra  impetus  the  NAO  report  generated 
within  the  Department.  The  final  savings  are  detailed 
below. 

Brief  details  of  external 
validation: 

Savings  are  derived  from  the  DE&S  Power  Steering 
database,  and  have  been  internally  audited  by  the 
MOD. 

In  terms  of  linking  the  NAO  report  to  the  savings 
achieved  by  the  Department,  RAND  Europe  conducted 
a  series  of  interviews  with  key  DE&S  personnel  to  verify 
the  progress  made  by  the  Department  against  the 
original  NAO  recommendations. 

Name  and  grade  of  person 
agreeing  impact  externally 

Georgina  Benzies,  Director  (Chief  Management 
Accounting) 

Value  of  Impact  (£) 

£29.2  million 

(£1.4  million  RDEL;  £27.7  million  IRDEL; 
£0.1  million  CDEL) 

Is  this  a  one-off  impact? 

Yes  (although  there  may  be  scope  to  examine  in  future 
whether  some  of  these  savings  are  recurring) 

Is  this  the  first  year  of  claiming 
impact? 

Yes 

If  2nd  or  subsequent  year,  when 
was  the  impact  first  claimed? 

N/A 
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How  many  years  in  total  have 
you  agreed  to  claim  this 
impact? 

One 

If  this  impact  will  be  claimed 
again  next  year  what  will  be 
the  likely  value? 

N/A 
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